HDTV editing and effects previsualization using SDTV devices

ABSTRACT

A system provides real-time previsualization of effects to be added to high definition (HD) video data and real-time rendering of the HD video data including the added effects. The computer based system for editing high definition television (HDTV) resolution video includes a high definition video system connected to a standard definition video system and a high definition storage system. A resizer reformats the high definition video data to standard definition resolution for real-time processing and previsualization.

FIELD OF THE INVENTION

The present invention is related to editing high definition video data.

BACKGROUND OF THE INVENTION

Currently, standard definition television (SDTV) resolution editorsexist which are used for editing video data. It is possible to create acomposition and view the editing effect in real time. However, forediting high definition television (HDTV) resolution video data, highdefinition (HD) editing equipment may be cost prohibitive and HD videodata that is edited is often stored back to a storage system each timean edit is made and the edited version of the HD video data is viewedonly after the effects are made. When more edits are needed, the HDvideo data is sent to an editor, edited and saved again to the storagesystem. The edited version is then available to be viewed. This processis expensive and time consuming.

With the advances in high definition television (HDTV) resolutionsystems and the uncertainties in future standards, a need exists forediting HDTV video data cost effectively. Current SDTV systems do notallow editing or digital video manipulation of original HDTV compressedvideo data and the ability to display the edited HDTV video data in fullresolution.

SUMMARY OF THE INVENTION

The present invention provides a system for real-time previsualizationof effects to be added to high definition (HD) video data and real-timerendering of the HD video data including the added effects. The systemincludes a resizer for reformatting the HD video data to fit within thebandwidth limits of standard definition (SD) equipment.

Accordingly, one aspect is a system for editing high definition videodata. A random-access, computer-readable and re-writeable storage systemstores high definition video data in data files. A high definition videosystem includes a high definition video data router for receiving highdefinition video data from the storage system and directing the highdefinition video data to a first and a second output. A resizer isconnected to the first output of the router and has an output providingstandard definition resolution video data based on the high definitionvideo data. A high definition output module is connected to the secondoutput of the router. A standard definition video editing systemincludes a standard definition digital video effects module having aninput for receiving the output of the resizer and a display, such as astandard definition monitor or computer monitor for previsualizing videodata with the added effects which is output from the standard definitiondigital video effects module at an high definition frame rate.

Another aspect is a method for editing high definition video data usingstandard definition video equipment. High definition video data isreceived and resized to fit the bandwidth of the standard definitionvideo equipment. Effects are added to the resized high definition videodata in real-time using the standard definition video equipment and theresized high definition video data including the added effects isprevisualized on standard definition video equipment. Full resolutionhigh definition video data with the added effects is rendered.

Another aspect is using a high definition television resolution monitorfor full resolution viewing of the edited data at high definition framerates.

Another aspect is using a multiformat router as the high definitionvideo data router.

Another aspect is saving an original copy of high definition video datain a data file and resizing the high definition video data whileretaining the original high definition video data unchanged in memory.

Another aspect is storing the result of rendering full resolution highdefinition video data with added effects in a data file.

BRIEF DESCRIPTION OF THE DRAWING

In the drawing,

FIG. 1a is a block diagram of a high definition television (HDTV)resolution editing system;

FIG. 1b is a resizer according to an embodiment;

FIGS. 1c-1 f show modifications to the editing system of FIG. 1a;

FIG. 2 is a block diagram of a standard definition television (SDTV)editing system used for HDTV editing and digital visual effectsprevisualization;

FIG. 3 is a block diagram of a SDTV editing system directly interfacingwith a HDTV resizer, according to an embodiment;

FIG. 4 is a flowchart describing how the SDTV editing system of FIG. 2is used for HDTV editing according to an embodiment;

FIG. 5 is a block diagram of a fully featured real-time HDTV editingsystem according to an embodiment;

FIG. 5a is a block diagram of a modified system of FIG. 5;

FIG. 6 is a block diagram of a real-time uncompressed three stream HDTVvideo system using an accelerated graphics port;

FIG. 6a is a block diagram of a modified system of FIG. 6; and

FIG. 7 is a block diagram of an HDTV video system.

DETAILED DESCRIPTION

FIG. 1a is a block diagram of one embodiment of a system for editinghigh definition television (HDTV) resolution video data with a singlestream of video data to be edited. FIG. 1a includes a high definition(HD) video system 104 connected by bus 144 to an HD storage system 102.HD video system 104 is also connected by bus 145 to a standarddefinition television (SDTV) resolution frame buffer 126, which in turnis connected to a display 138, which may be progressive or interlacedformat, such as a computer display or an standard definition (SD)monitor. The data displayed is related to the format of the HD videodata (e.g., if the HD data is in progressive format, then the SD displayis a progressive format display). HDTV video input/output module 140 isconnected to an output of router 120 and provides an output to HDTVmonitor 142. Another editing system may be found in U.S. PatentApplication entitled “EDITING SYSTEM WITH ROUTER FOR CONNECTION TO HDTVCIRCUITRY” by Morton Tarr et al. filed Apr. 3, 1998.

In order for a digital HDTV video signal to fit in the transmissionbandwidth of standard systems (e.g., systems conforming to NationalTelevision Systems Committee (NTSC) or Sequential Color and Memory(SECAM) standards), the HDTV video signal may typically be compressedusing a ratio of about 5:1. For example, the NTSC standard requires anaspect ratio (ratio of width to height) of 4:3 with 525 scanning linesper picture and a television signal bandwidth of 6 MHz. The SECAMstandard specifies 625 scanning lines per picture and a televisionsignal bandwidth of 8 MHz. SDTV resolution video formats are typically704-pixel-by-480-line or 640-pixel-by-480-line. In contrast,HDTV-resolution video data typically requires 30 MHz of bandwidth,doubles the number of scan lines in a frame and changes the aspect ratioto 16:9. HDTV-resolution video is typically 1920-pixel-by-1080-line,although it may be in other formats, such as 1280-pixel-by-720-line inprogressive or interlaced formats. HD video data may include any data ata resolution higher than SD video data, such as, for example, data witha resolution greater than 525 scan lines having a component rate of morethan 30 frames/second, with 8 or 10-bit precision. HD data may beinterlaced or progressively scanned and the present invention is notlimited to either format.

In an embodiment of the present invention, the single stream editingsystem of FIG. 1a can operate in real-time to perform cut edits byrandomly sequencing a non-linear, high definition storage system 102 tooutput data in an order indicated by a desired video sequence. Thestorage system 102 may be, for example, a randomly accessible diskarray.

The HD video system 104 of FIG. 1a includes an HD video data router 120for receiving video data and directing video data to HD videoinput/output module 140, to frame buffer 122 or to resizer 124. Resizer124 adjusts the higher resolution data to a lower resolution format. Forexample, resizer 124 reformats HD video data to provide an output whichfits the bandwidth of SDTV equipment which can typically process datawith a resolution less than 525 scan lines having a component rate ofless than 30 frames/second. The output of resizer 124 is sent to SDTVframe buffer 126. Resizer 124 reformats the HDTV-video data to allow anSDTV representation to be displayed on SDTV equipment such as an NTSCmonitor or RGB computer monitor 138. Before resizer 124 reformats thedata, the data is low-pass filtered to avoid aliasing the data. Resizer124 then generates only desired output data.

FIG. 1b illustrates an example resizer. The resizer shown in FIG. 1boperates on the luma component of the HD video data. A similar resizercircuit may be used to resize the chroma component of the HD video data.The demultiplexer 190 receives HD video data as an input. Demultiplexer190 separates a 148.5 MHz HD video stream into two 74.25 MHz streams,one of which contains the luma components and the other of whichcontains the chroma components. Field Programmable Gate Arrays (FPGA)may be used in the resizer when the data rate is 74.25 MHz, which iswithin the range of FPGA devices. In an alternate embodiment, anApplication Specific Integrated Circuit (ASIC) may be used.

The embodiment shown in FIG. 1b uses 5-tap filters to perform a 5:1resize operation. However, the number of filters taps can be changed toperform other resize operations. The resizer changes the data rate by aratio of 1920/720 to produce SDTV-compliant video. By changing thevalues stored in the vertical counter 196, horizontal counter 197, andthe coefficient stores 198, the resizer may scale the video to anyarbitrary size.

The luma component of the video is clocked into a first set of registers192 at the HD video rate, and then clocked into a second set ofregisters 193 at a rate determined by the horizontal resize factor. Thedesired horizontal pixel value is calculated by multiplying the fivecurrent coefficients by their respective input pixels, and summing theresults. The coefficients represent functions which determine theweighting given to each input pixel. After each new output pixel iscalculated, the input data is advanced by a value determined by thehorizontal counter, thereby presenting the appropriate data to calculatethe next pixel. For an operation which reduces an input image by a ratioN:1, the horizontal resize circuit may, on average, produce a new pixelevery N 74.25 MHz clock cycles.

The output of the horizontal resizer may be written to a FIFO 195 onselected clock cycles at the HD component clock rate of 74.25 MHz. Thedata is read from the FIFO 195 at the SD rate of 27 MHz and verticalresize operations are performed at the SD clock rate. The verticalresizer may operate similarly to the horizontal resizer, except thatresized lines are stored, rather than pixels, prior to calculating thenext output value.

Intermediate storage elements not shown in FIG. 1b may be used tofacilitate operation at the required clock rate. The coefficients andcounter values may be stored within the device, or they may be loaded bya host computer or other controlling device.

There are several kinds of systems (e.g., HD video system 104) that maybe used to author, process and display multimedia data. These systemsmay be used to modify the data, define different combinations of data,create new data and display data to a user. A variety of techniques areknown in the art for implementing these kinds of systems.

Multimedia authoring, processing and playback systems typically have adata structure which represents the multimedia composition. The datastructure ultimately refers to clips of source material, such asdigitized video or audio, using an identifier of the source material,such as a unique identifier or a file name, and possibly a temporalrange within the source material defining the clip. The identifier maybe of a type that may be used with a list of equivalent data files toidentify a file name for the source material. An index may be used totranslate the temporal range in the source into a range of bytes withina corresponding file. This range of bytes may be used with the segmenttable for the file to identify segments of data that are needed and thestorage units from which the data is retrieved.

A list structure may be used to represent part of a multimediacomposition. Clips may include a reference to a source identifier and arange within the list. Generally, there may be such a list for eachtrack of media in a temporal composition. There are a variety of datastructures which may be used to represent a composition. In addition toa list structure, a more complex structure is shown in PCT PublishedApplication WO93/21636 published on Oct. 28, 1993. Other examplerepresentations of multimedia compositions include those defined by OpenMedia Framework Interchange Specification from Avid Technology, Inc.,Advanced Authoring Format from the Multimedia Task Force, QuickTime fromApple Computer, DirectShow from Microsoft, and Bento also from AppleComputer, and as shown in PCT Publication W096/26600.

The data structure described above and used to represent multimediaprograms may use multiple types of data that are synchronized anddisplayed. The most common example is a television program or filmproduction which includes motion video (often two or more streams ortracks) with associated audio (often four or more streams or tracks).

Because the video and audio data may be stored in different data filesand may be combined arbitrarily, better performance may be obtained ifrequests for data for these different data files are managedefficiently. For example, an application may identify a stream for whichdata can be read, and then may determine an amount of data which shouldbe read, if any. A process for performing this kind of management ofread operations is shown in U.S. Pat. No. 5,045,940. In general, anapplication determines which stream has the least amount of dataavailable for display. If there is a sufficient amount of memory fordata to be played back for that stream to efficiently read an amount ofdata, then that data is read from the file. When it is determined thatdata for a stream should be requested, each segment of the data isrequested from a storage unit selected from those on which the segmentis stored. In order to identify which files to request from the storageunit, the editing system may convert a data structure representing acomposition, into file names and ranges within those files.

In FIG. 1a, disk buffer memory 114 may be, for example, a circularbuffer and it receives a sequence of digital still image. Each stillimage may represent a single frame, i.e., two fields, or a single fieldof motion video data from HD storage system 102 which is to be processedto render edited video data. Application software such as, for example,software used in the Media Composer editing system by Avid Technology,Inc., may play the sequence of digital still images to be manipulatedinto the buffer memory 114. Disk buffer memory 114 may hold multipleframes of video to be sent through one or more coder/decoder processors(codecs) 116 to router 120 to reduce overhead when linear devices areused with the non-linear access storage system 102.

Display 138, such as an SDTV monitor or computer display, is connectedto SDTV frame buffer 126 and is used for previsualizing HDTV compositeswhich have been processed to include user specified edits. HDTV monitor142 receives an output from HDTV video I/O module 140 and is used forviewing full resolution rendered HDTV composites which include the editsprevisualized on the SDTV equipment.

A hardware dataflow interface which enables asynchronous data processingelements to be interconnected using an interconnection protocol thatcontrols the flow of data between processing elements may be used tocontrol the flow of data between elements in FIGS. 1a, 1 c-3 and 5-7.Such an interface is described in U.S. patent application Ser. No.08/879,981 filed Jun. 20, 1997, U.S. Patent entitled “APPARATUS ANDMETHOD FOR CONTROLLING TRANSFER OF DATA BETWEEN AND PROCESSING OF DATABY INTERCONNECTED DATA PROCESSING ELEMENTS” by Craig R. Frink filed Apr.3, 1998, U.S. Patent Application entitled “A PACKET PROTOCOL FORENCODING AND DECODING VIDEO DATA AND DATAFLOW SIGNALS AND DEVICES FORIMPLEMENTING THE PACKET PROTOCOL” by Craig R. Frink et al. filed Apr. 3,1998, and U.S. Patent Application entitled “METHOD AND APPARATUS FORCONTROLLING DATA FLOW BETWEEN DEVICES CONNECTED BY A MEMORY” by Craig R.Frink which are incorporated herein by reference. The flow controlallows the processing elements to be media and format independent. Thesystem is not limited to flow control and other interfaces and methodsfor synchronization may also be used. For example, in FIG. 1a, adataflow interface may be used to control the flow of data betweenstorage system 102 and HD video system 104 or data between HD videosystem 104 and SDTV frame buffer 126.

The system shown in FIG. 1a may be modified to include an external HDTVcodec 160 as shown in FIG. 1c, with for example, Panasonic's HD-D5 codecfor tape-based HDTV video storage. Using this codec in the non-linearediting (NLE) application shown in FIG. 1a allows format compatibilitywith Panasonic HDTV recording methods, and may be used to compress anddecompress HDTV video that is too large to fit within the limits of anediting system. The Panasonic HDTV codec can replace HD codec 116 and isconnected to the disk buffer memory 114 and HDTV router 120 usingconventional methods (such as Society of Motion Picture and TelevisionEngineers (SMPTE) standard interfaces 259M and 292M). This method ofconnectivity applies equally to other devices, including the video datarouter, video I/O, and other codec devices (Sony HDCam, MPEG2 HL @ PP,etc.).

The real-time single stream HDTV editing system shown in FIG. 1c may usesignificant rendering time when using digital video effects (DVEs) thatinvolve multiple streams (dissolve, super-imposition, positioning of akeyed image, etc.). The DVEs added using the HDTV editing system of FIG.1c can be viewed once they are rendered but are not generally able to beviewed when being constructed.

In one embodiment, the rendering time can be reduced by modifying thesystem of FIG. 1a through the addition of a hardware DVE module 150, asshown in FIG. 1d. The hardware DVE module 150 which is connected to HDTVvideo data router 120 and is used to accelerate video manipulationoperations, such as, for example, 3D DVEs or 2D resizing. In thisembodiment, the editing system includes a hardware DVE module and secondframe buffer, similar to frame buffer 122, for storing an alpha channel,in addition to the video, and to provide connectivity bandwidth forsimultaneous access of two streams (i.e., one writing and one reading).This system operates similarly to the system of FIG. 1a, but instead ofreading the frame buffer contents using software when rendering a DVE,the system plays one stream of video into one of the frame buffers(e.g., 122), and then plays a second stream directly to the DVE module150. The video stored in the frame buffer (e.g., 122) is read in concertwith the video playing from disk (e.g., 108), and both streams passthrough the DVE module together. The resulting video is stored to theadded second frame buffer and the resulting video may then become thesource video for a subsequent DVE, or stored back to disk.

In another modification of FIG. 1a shown in FIG. 1e, the addition of asecond video channel from the storage system 102 enables the playing ofmultiple video streams concurrently for real-time DVEs and allows theDVE operations to be visualized in real-time. This includes adding anadditional disk buffer 114, an additional HD codec 116 and an additionalHD-1080 frame buffer 122 to the system of FIG. 1a. The dual streamsystem uses dual HDTV data rate and resolution frame buffers to capturevideo and an alpha channel when rendering more than two streams of video(the ability to store the alpha channel in the rendered video allowsforegrounds to be composited, rather than rendering backgrounds only).The frame buffers also provide a mechanism for frame delay compensationwhen switching elements in and out of the video pipeline which connectsthe video devices in the system.

The HDTV video devices shown in FIG. 1a may be separate components (suchas when using a Panasonic HD-D5 codec), may be external to a computer(as is common when using conventional linear video equipment), or may becombined into a single design (as is common in most computer-based NLEequipment).

In one embodiment, the modified system of FIG. 1a just described, whichincludes a second video channel from storage system 102, may use nonreal-time hardware assisted rendering when the number of streams thatare combined in a composition increases beyond the real-time capabilityof the system (two streams in the above system). For example, in thesystem shown in FIG. 1f, in a three layer composition (three concurrentvideo streams) the first two streams may be combined and the resultstored in a frame buffer. Multiple frames in a sequence can be processedfor efficiency. The third stream is added to the intermediate resultstored in the HDTV frame buffer (e.g., 114) by using only one of thevideo stream channels from the storage system (e.g., 102). The resultingvideo from the final compositing step (combining the third stream withthe result of the first two streams) can then be stored back to disk(e.g., 108) using the second video stream channel, or rendered again toa frame buffer (e.g., 114). The video (and alpha) channel may also bestored compressed or uncompressed to disk. The system is able to storevideo with its associated alpha channel, when one is generated, forcombination with foreground streams. Backgrounds may also be stored toreduce the overhead of storing alpha streams to disk.

The modified system described above provides real-time performance andimproves rendering time. However, non real-time rendering is often usedsince the number of codecs used in the system may limit real-timeoperation. The addition of a third video channel allows two streams ofvideo to be played while one is recorded. Intermediate composites can bestored to disk in real-time for immediate application as more layers ofvideo (equating to more streams) are combined. The video and associatedalpha stream (compressed using run length encoding) may be storedtogether for simplicity, or may be stored separately. A tape basedstorage device may be used to capture the video once an effect iscreated. In addition, an additional video channel (and HD codec) enablesthe video to be stored in real-time to disk for use in a subsequentcomposition.

FIG. 2 is a block diagram of a SDTV editing system used for real-timeHDTV editing and digital visual effects previsualization according to anembodiment of the present invention. This system allows lower costediting by using available SDTV devices to previsualize HDTV videoeffects and allows video sequences to be edited while preserving thevideo that is stored in storage system 202 in its original HDTV format(e.g., uncompressed, HD-D5, HDCam, MPEG2 HL@PP, etc.). In addition, thesystem shown in FIG. 2 is compatible with fully featured real-time HDTVsystems.

The system shown in FIG. 2 includes a video data storage system 202, HDvideo system 204 and standard definition television non-linear editingsystem (SDTV NLE) 206. Storage system 202 includes non-linear storagedevices, such as disks 208 and stores HDTV video digitally usingcompressed and uncompressed formats. The video that is stored on disks208 is not modified in the non-linear editing process to avoidgeneration losses when using compression. Therefore, when a stored videoframe is manipulated, the original frame remains untouched and a newframe or sequence of frames is created for the new video. The storagesystem 202 is capable of playing multiple streams of full resolution andfull data rate HDTV video stored in compressed or uncompressed formatsfor the purpose of generating real-time transition effects (e.g., wipes,dissolves, picture-in-picture, etc.). Storage system 202 also includesdisk data router 210 and disk controllers 212 and is scalable dependingon the number of streams in the editing system.

HD video system 204 is connected between HD storage system 202 andnon-linear editing system 206. Video system 204 includes HD disk buffers214 which synchronize the video data that is passed between the storagesystem 202 and the HD router 220 through HD codec 216. HD codec 216 isused to decompress HD video data that was previously compressed forstorage before it is sent to non-linear editing system 206. HD codec 216may also be used to compress or decompress HD video data for storage.HDTV video data router 220 determines whether HD video data is to besent to the non-linear editing system 206, to HD storage system 202 oroutput to HDTV video I/O 240.

Resizers 224 are used to take the HD video data from HD video datarouter 220 and reformat it to fit into the bandwidth requirements of thenon-linear editing system through video filtering and resampling toscale the spatial resolution of the HDTV video. Resizers 224 convert HDvideo data to SDTV resolution in real-time. This allows an SDTVrepresentation of an HDTV image to be displayed on a standard NTSCmonitor or on a RGB computer monitor. The HDTV video is able to retainits original aspect ratio (16:9 or 4:3) and the SDTV system is able tooperate using progressive and interlaced formats in these aspect ratios.

FIG. 2 includes an SDTV frame buffer 226 between the HD video 204 andthe SDTV video effects module. Hardware which allows systemresponsiveness to be high and latency through the SDTV frame buffer 226to be low may be used to control the process of filling the SDTV framebuffer 226. Software may also be used. The HDTV video system 204generates the SDTV data at a constant rate, while the data rate of theSDTV pipeline may be variable (in a dataflow system). However, theaverage data rate of the constant flow data is approximately 21 MHz. TheSDTV pipeline uses a valid signal as part of the processing pipeline andinterconnection to manage the reception of data from the frame buffer222. Software such as the software used in the Media Composer editingsystem by Avid Technology, Inc. may manage the frame buffer 222 throughthe use of circular link list direct memory access (DMA) structures.Since the data rate of the video arriving at the frame buffer 222 isconstant, after the video flow commences, software such as the softwareused in the Media Composer editing system by Avid Technology, Inc. isable to track the state of the SDTV frame buffer pointers to avoidoverrunning the buffer, and to avoid reading the buffer when data is notyet valid.

Non-linear editing system 206 is used for previsualization of edited HDvideo data in real-time. SDTV digital video effects module 232 addsdesired transition effects (e.g., wipes, dissolves, picture-in-picture,etc.) to the digital video data or manipulates the digital video data(e.g., page curls, ripples, etc.). The video data including the effectsis blended by the SDTV effects module 232, which outputs the editedvideo data to SDTV video input/output module 234. The video data withthe effects may be previsualized on SDTV monitor 236 or on a computerdisplay 238. Previsualization includes real-time viewing, at less thanfull resolution, of an effect applied to video data. Previsualizationsaves time since effects can be viewed as they are added to the videodata, without saving the edited video data to a tape or disk andretrieving the same or a new piece of video data to be edited each timea new effect is desired. Since SD equipment may be used for addingeffects and HD equipment is not needed for previsualization, the cost ofediting may be reduced.

FIG. 3 shows an alternative embodiment of the SDTV editing system ofFIG. 2. In FIG. 3, SDTV frame buffers (e.g., 226, FIG. 2) are not usedand the SDTV NLE system 306 is connected directly to the resizer 324. Inthis embodiment, the HDTV frame buffer 322 and the HDTV video router 320control the flow of data. This is possible using hardware with directmemory access and flow control capability. In this system, separateinterfaces to read and write to memory may be used. Three alternativemethods which may be used to render HDTV video data to full resolutionusing the system in FIG. 3 are described below.

In one method of rendering using the system shown in FIG. 3, the SDTVprevisualization system 306 renders HDTV video data. The HD framebuffers 322 capture the HDTV video which is retrieved from storagesystem 302 and editing effects are processed in non real-time. The framebuffers 322 store a plurality of HDTV frames to assist in capturing andstoring frames to disk.

In another method using the system shown in FIG. 3, rendering the HDTVvideo to full resolution may also be performed by using the framebuffers 322 and real-time HDTV video devices. Multiple uncompressedframes of HDTV video may be rendered using the frame buffers 322 astemporary storage for the uncompressed intermediate composites. This ispossible by decompressing the frames, creating the effect, storing theintermediate result in the frame buffer 322 and then adding additionallayers in combination with the stored uncompressed HD video. The framebuffer 322 memory size determines the limit on the number ofintermediate frames in a single rendered composition.

In another method using the system shown in FIG. 3, rendering the HDTVvideo to full resolution may include using non real-time hardware thatis able to process the full resolution video data, but not in real-time.The frame buffers decouple the real-time instantaneous HDTV data ratefrom the lower data rate of the digital video effect.

The operation of FIGS. 2 will now be described in connection with theflowchart of FIG. 4. In step 405, HDTV video data is captured by HDTVvideo data router 220 which sends the HDTV video data to be stored todisk 208 in the HD storage system in step 410. The stored HD video datamay be compressed or uncompressed. The HDTV video data may be capturedin real-time from a video source (e.g., tape deck, camera, etc.) orthrough a computer-based method (e.g., a Local Area Network (LAN),Digital Linear Tape (DLT), etc.). In step 415, HDTV video data router220 retrieves video data from the HD storage system 202. Video datarouter 220 may retrieve a frame of video data or a non-linear sequenceof frames for editing. The HDTV video data may be compressed ordecompressed by HD codec 216 before it is sent to be stored orretrieved. HD disk buffer memory 214 controls the flow of the video databy exchanging control flow information with disk controller 212. Handshaking protocols or other interfaces, such as the hardware dataflowinterface described above, may be used to control the flow of data.

Resizer 224 provides an output of SDTV resolution video data based onthe HD video data by adjusting the HDTV video data to fit into the SDTVbandwidth of the SDTV NLE system 206. As discussed above, in order forthe digital HDTV signal to fit in the standard transmission bandwidththere is a need for a compression ratio of approximately 5½:1.

The resized HDTV video data is sent to SDTV frame buffer 226 which isused to synchronize the transfer of video data in the SDTV editingsystem 206 in step 425. The SDTV editing system 206 adds effects in step430 to the resized HDTV video data. The effects may include transitioneffects such as wipes, fades, etc.

In step 435, the resized HDTV video data, including the added effects,is previsualized on SDTV monitor 236, on computer display 238, or otherdisplays. This allows the user to view the added effects from editing ata lower resolution and avoids rendering new video for each effect aswell as the problems associated with incurring a generation loss fromcompression of the video data.

In step 440, when more video data is needed for editing, the processreturns to step 415, in which the router retrieves video data from theHD storage system 202. When more video data is not needed for editing,then the previsualized video data is rendered to full resolution in step445 by SDTV effects module 232 in non real-time, using hardware orsoftware rendering devices as described above.

In step 450, the rendered video is typically stored in storage system202, and can then be retrieved for full resolution viewing on an HDdevice such as HDTV monitor 242, in step 455. The stored video data mayalso be output for creating video for distribution.

The operation of FIG. 3 is similar to the operation of FIG. 2 describedabove in reference to FIG. 4. However, in step 425, if the SDTV NLEsystem (i.e., 306) does not include SDTV frame buffers (e.g., 226, FIG.2) then the resized HDTV video data from resizer 324 is sent directly toSDTV editing system 306, and the transfer of the video data iscontrolled by HD-1080 frame buffers 322.

FIG. 5 is a block diagram of a fully featured real-time HDTV editingsystem. The system in FIG. 5 is comparable to an SDTV editing system,but uses HDTV video. The system includes a real-time HD DVE module 554for each video channel (color correction, resize, flop, image crop,etc.), and more complex 3D digital video effects capability fortransition effects and video manipulation (page curls, ripples,perspective warp, etc.) by using HDTV 3D DVE module 550. The system isable to composite multiple video streams using the 3D DVE module 550,and also using HDTV video router 520, which includes a mixer, andDownstream Keying (DSK) hardware. The system also includes DSK hardwareto combine real-time video with a static graphic (e.g., a title), aswell as an animation rendered in real-time using for example, softwaresuch as a graphics accelerator 552 which produces a video output with akey, including products such as OpenGL 3D graphics DVE accelerator bySilicon Graphics. HDTV video compression is possible using HD codecs516, although uncompressed video playback is also possible usinguncompressed video input/output paths 518. The system shown in FIG. 5includes three video channels for real-time rendering while playing.

FIG. 5 operates in real-time using HDTV resolutions and data rates. Inaddition to processing HDTV video, the system in FIG. 5 is also capableof processing SDTV video in real-time, using the same components. Thesystem is able to route SDTV video through the HDTV router 520 (singleand multiple streams), and in combination with HDTV video. The DVE,mixing, and other functions scale HD video data from HDTV-resolution tothe lower resolution SDTV processing and data rates.

A modification to FIG. 5, as shown in FIG. 5a, may be made to the HDTV3D DVE module 550 which results in a hybrid system using an SDTV 3D DVEmodule 572 and HD-to-SD resizers 570 and SD-to-HD resizers 574 toreplace the HDTV 3D DVE module 550. This modified system may be used toprevisualize the 3D DVE module in real time, using SDTV resolutions.

FIG. 6 is a block diagram of a real-time uncompressed three streamreal-time HDTV video system using an Accelerated Graphics Port (AGP)interface for a bus protocol to a peripheral connect interface (PCI)bus, for example, between the HD video system 604 and the computermemory 652.

The system of FIG. 6 uses a plurality of PCI computer busses, forexample, FibreChannel's PCI interfaces, to isolate HDTV data streams(64-bit PCI or 66 MHz PCI used for real-time HDTV uncompressed formats)for the purpose of aggregating the video streams into a common highspeed host computer memory. This enables disk data to be sent across PCIbusses, transferred to host memory 652, and then transferred over thehigh performance data interface (AGP) 654 to HD video processing system604. The system 604 uses multiple DMA channels at the AGP interface 654to access the host memory 652 for playing multiple streams of video, andfor capturing multiple streams of video. The AGP DMA device may beimplemented as separate DMA devices or as a DMA capable of multiplesimultaneous contexts. The PCI bus in an embodiment may be able tosustain a component bandwidth of 1 or 2 streams of HDTV compressed videodata, with approximately 30-40 Megabytes per stream. The AGP may sustainthe bandwidth of uncompressed HDTV video data.

The AGP interface 654 and high speed host memory 652 may also be used tocreate connectivity between the storage system 602 and the HD videosystem 604 for rendering software DVEs. The HD video system 604 may alsoplace video frame buffers in host CPU memory 652, such as those requiredfor rendering, timebase correction, and DVE operations requiring inputand/or output frame buffers, etc., to reduce the cost of the videosystem hardware.

The HDTV video system 604 includes disk data buffers 652 (to buffer datafrom the disk controllers 612), frame buffers 614 to buffer latencies inaccessing disk data buffers 652, an interface to an HD codec (not shown)(held internally on a PCI board or externally using a digitalinterconnect, such as, for example, Panasonic's HD-D5 codec), and aninterconnect to the HDTV video I/O 640 and monitor 642. Full resolutionHDTV frame buffers 622 operate at a full data rate to capture HDTVframes when necessary for full resolution effects processing(rendering). Video data is transferred between the computing system andthe HD frame buffers 622 and resizer 624 subsamples the full resolutionHDTV video to SDTV data rates, in the 16:9 aspect ratio (or 4:3). Theresized video is transferred to a display 638 (i.e., a monitor) and anuncompressed SDTV video editing system (not shown), as discussed above,for real-time DVE and editing previsualization.

In systems limited by PCI and storage throughput constraints and whenthe number of uncompressed storage elements is cost prohibitive, datacompression is needed. The storage systems of an embodiment of thepresent invention may be independent of the system as long as a methodexists to distribute the video bandwidth to the HDTV devices, whileproviding symmetric data access. Multiple storage controllers andsegmented PCI busses may be used when compressing HDTV video or whenusing uncompressed HDTV video.

The system of FIG. 6a is a block diagram of a real-time three streamuncompressed HDTV video system. The number of PCI interfaces scales withthe bandwidth demand, as does the number of separate PCI bus segments tosupport the bandwidth of the system. The system shown in FIG. 6a usesseparate PCI bus segments for each HDTV uncompressed disk data buffer614. The storage system 602 provides a shared data access method byrouting disk data packets to the appropriate disk controller for theHDTV stream that the data is directed toward. The HDTV router 620 mayprovide connectivity of video streams when the video manipulation occursin a device such as DVE module 650.

FIG. 7 is a block diagram of an HDTV video system. Each video channelinterface shown in FIG. 7 is implemented in a single ASIC. Each ASIC mayinclude a 64-bit PCI interface (741), 1500 MB/s memory interface for thedisk data buffer 714, and the HD DVE module 754. The 2D DVE operationsmay include color correction, chroma, and luma key generation, videoresizing, and motion effects. Each video channel interface is able toprocess video in real-time and interface to the HDTV data router ASICusing a high speed, desktop video interconnect. Dual HDTV frame bufferswith PCI interfaces may be controlled by a single ASIC. Module 706 maybe used for viewing HDTV composites.

One advantage of the present invention includes the ability to remain“on-line” while performing edits and viewing the edited video, withoutgoing back to a disk or tape to retrieve a piece of video. Anotheradvantage includes reducing the number of real-time HDTV componentsrequired when editing HDTV video by using standard definition equipment.The present invention allows HD and SD formatted video data to becombined and lower resolution devices can be used to effectivelyprevisualize effects in real-time even though the pre-visualized imageis not at full resolution. In addition, to prevent degradation, anoriginal video data file is stored in a storage system and editing isperformed on a copy of the video data file.

The editing system of the present invention may use dataflow videopipelines, interconnects, and DMA controllers, coupled through FIFOs, toenable the combination of HDTV and SDTV video devices and data rates inthe same system. When locking to an HDTV reference clock, the SDTVsystem is only required to remain in synch to output frames as theybecome available from the HD subsystem. In the case of creating videofor SDTV distribution from an HDTV source, and using external HDTVdevices in conjunction with this invention, the 27 MHz clock referencefrom the SDTV system generates a 74.25 MHz clock for the HDTV subsystemand its external components.

Having now described a few embodiments, it should be apparent to thoseskilled in the art that the foregoing is merely illustrative and notlimiting, having been presented by way of example only. Numerousmodifications and other embodiments are within the scope of one ofordinary skill in the art and are contemplated as falling within thescope of the invention.

What is claimed is:
 1. A non-linear system for editing of highdefinition video data comprising: a random-access, computer readable andre-writeable storage system for storing high definition video data indata files; a high definition video system including: a high definitionvideo data router for receiving high definition video data from thestorage system and directing the high definition video data to a firstand a second output; a resizer connected to the first output of therouter and having an output providing standard definition resolutionvideo data based on the high definition video data; a display forreceiving the standard definition resolution video data from the outputof the resizer; at least one three-dimensional digital video effectsmodule connected to the high definition router; at least threecoder/decoder processors connected between the storage system and thehigh definition video data router, wherein the video data routerreceives three streams of high definition video data from the storagesystem; and at least one digital video effects module per streamconnected between the coder/decoder processor and the high definitionvideo data router.
 2. The non-linear storage system of claim 1, whereinthe at least one three-dimensional digital video effects moduleincludes: at least two high definition-to-standard definition resizers;at least two standard definition-to-high definition resizers; and astandard definition three-dimensional digital video effects module.
 3. Anon-linear system for editing of high definition video data comprising:a random-access, computer-readable and re-writeable storage system forstoring high definition video data in data files; a high definitionvideo system including: a high definition video data router forreceiving high definition video data from the storage system anddirecting the high definition video data to a first and a second output;a resizer connected to the first output of the router and having anoutput providing standard definition resolution video data based on thehigh definition video data; and a high definition output moduleconnected to the second output of the router; and a standard definitionvideo editing system including: a standard definition digital videoeffects module having an input for receiving the output of the resizer;and a standard definition monitor for previsualizing video data with theadded effects which is output from the standard definition digital videoeffects module.
 4. The non-linear system as recited in claim 3, whereinthe high definition output module is connected to a high definitiontelevision resolution monitor for full resolution viewing of the editedvideo data.
 5. The non-linear system as recited in claim 3, wherein thehigh definition video data router is a multiformat router.
 6. Thenon-linear system as recited in claim 3, wherein the router receives twostreams of data.
 7. The non-linear system as recited in claim 6, furtherincluding at least two resizers.
 8. The non-linear system as recited inclaim 3, further including digital video effects equipment receiving anoutput of the resizer.
 9. The non-linear system as recited in claim 3,wherein the standard definition digital video effects module receivesthe output of the resizer at the rate of the high definition video data.10. The non-linear system as recited in claim 3, wherein the standarddefinition video editing system includes a standard definition buffer toreceive the output of the resizer.
 11. The non-linear system as recitedin claim 3, wherein the high definition video system includes a highdefinition coder/decoder processor.
 12. A method for editing highdefinition video data using standard definition video equipmentcomprising the steps of: receiving high definition video data; resizingthe high definition video data to fit the bandwidth of the standarddefinition video equipment; adding effects to the resized highdefinition video data in real-time using the standard definition videoequipment; previsualizing the resized high definition video data inreal-time including the added effects on standard definition videoequipment; and rendering full resolution high definition video data withthe added effects.
 13. The method of claim 12, wherein the resizing stepfurther includes: saving an original copy of the high definition videodata in a data file; performing the resizing on a copy of the highdefinition video data.
 14. The method of claim 12, further includingstoring the result of the rendering step in a data file.
 15. A resizerfor reformatting high definition video data to fit within bandwidthlimits of a lower definition system including: a demultiplexer forseparating components of high definition video data, the demultiplexerhaving an input for receiving high definition video data and an outputfor sending separated components of the high definition video data; afirst set of registers having an input for receiving an output of thedemultiplexer and an output for sending data at a rate determined by aresize factor for reformatting the high definition video data; a secondset of registers having an input for receiving an output of the firstset of registers at the rate determined by the resize factor and havingan output for sending data at the high definition data rate; and a firstbuffer having an input for receiving an output of the second set ofregisters at a high definition data rate and an output for sending dataat a lower resolution data rate; a second buffer having an input forreceiving the output of the data at the lower resolution data rate fromthe first buffer and outputting resized video data.
 16. The resizer ofclaim 15, wherein the components of the high definition video data areluma and chroma components.
 17. The resizer of claim 15, furtherincluding: a resizer for operating on a chroma component of the highdefinition video data; and a resizer for operating on a luma componentof the high definition video data.
 18. The resizer of claim 15, whereinthe first and second buffers are a first-in-first-out buffer.
 19. Anon-linear video editing system comprising: a random-access,computer-readable and re-writeable storage medium for storing video datain data files, wherein the video data defines images having greater than525 scan lines and having a component rate of more than 30frames/second; a non-linear editor for defining a video program as asequence of portions of the data files, wherein each portion of a datafile is defined by a reference to the data file and a range within thedata file and for defining one or more effects; means for reading thevideo data from the storage medium according to the defined videoprogram; at least one resizer having an input for receiving the videodata read from the storage medium to provide output video data having aspatial resolution of less than or equal to 525 scan lines and having acomponent rate of less than or equal to 30 frames/second; at least onevideo effects module having an input for receiving the output of theresizer and an output providing edited video data in real time accordingto the one or more effects defined for the video program; and a displayconnected to the output of the at least one video effects module forprevisualizing the edited resized video data.
 20. The system of claim19, wherein the display is a progressive scan display.
 21. The system ofclaim 19, wherein the display is an interlaced format display.
 22. Thesystem of claim 19, further comprising: a high definition video systemincluding: a high definition video data router for receiving highdefinition video data from the storage medium and directing the highdefinition video data to a first and a second output wherein the resizeris connected to the first output and a high definition display isconnected to the second output.
 23. The system of claim 22, wherein thehigh definition video data router is a multiformat router.
 24. Thesystem of claim 22, wherein the high definition video router receivestwo streams of data.
 25. The system of claim 24, wherein the at leastone resizer is at least a plurality of resizers.
 26. The system of claim19, wherein the video effects module receives the output of the resizerat the rate of the high definition video data.
 27. The system of claim19, further comprising a standard definition buffer to receive theoutput of the resizer.
 28. The system of claim 22, wherein the highdefinition video system includes a high definition coder/decoderprocessor.
 29. A method for editing high definition video data usingstandard definition video equipment comprising the steps of: defining avideo program with nonlinear editor as a sequence of portions of datafiles, wherein each portion of a data file is defined by a reference tothe data file and a range within the data file, and including one ormore effects; receiving high definition video data from data filesstored on a random-access computer readable and writeable storage mediumaccording to the defined video program; resizing the high definitionvideo data to fit the bandwidth of the standard definition videoequipment; adding effects to the resized high definition video data inreal-time using the standard definition video equipment according to thedefined video program; previsualizing the resized high definition videodata in real-time including the added effects on standard definitionvideo equipment; and rendering full resolution high definition videodata with the added effects.
 30. The method of claim 29, wherein theresizing step further includes: saving an original copy of the highdefinition video data in a data file; performing the resizing on a copyof the high definition video data.
 31. The method of claim 29, furtherincluding storing the result of the rendering step in a data file.